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1.0  EXECUTIVE  OVERVIEW 


The  Director  of  the  Computer  Software  Systems  Directorate  (CSSD)  requested  the 
Software  Technology  for  Adaptable,  Reliable  Systems  (STARS)  Joint  Program  Office 
(SJPO)  to  have  an  assessment  made  of  the  STARS  Program.  Three  concerns  prompted  the 
request:  (1)  the  Program  is  to  receive  $42  million  in  FY86,  (2)  the  perception  that  STARS 
is  not  effective,  and  (3)  the  absence  of  an  effective  top-down  plan.  An  assessment  was 
carried  out  by  a  panel  convened  during  September  and  October  1 985,  by  the  Institute  for 
Defense  Analyses  (IDA).  A  primary  basis  for  the  assessment  was  the  STARS  Program 
Plan  dated  19  September  1985.  This  is  the  Executive  Summary  of  the  findings  and 
recommendations  of  the  panel.  A  larger  companion  document.  Volume  II:  An  Assessment 
of  the  STARS  Program,  September-October  1985,  describes  these  in  greater  detail. 

The  panel  consisted  of  people  with  varying  levels  of  experience  with  STARS.  The 
panel  members  were:  Dr.  Elizabeth  Bailey,  Dr.  Richard  DeMillo,  Mr.  Herman  Fischer, 

Ms.  Audrey  Hook,  Dr.  John  Kramer,  Dr.  Thomas  Probert,  Mr.  Samuel  Redwine,  Dr. 
William  Riddle,  and  Dr.  Robert  Winner.  An  additional  group  was  briefed  on  the  progress 
and  findings  of  the  panel  on  September  26,  1985.  The  members  of  this  group  were  Mr. 
Joseph  Batz,  Dr.  Barry  Boehm,  Mr.  Bill  Carlson,  Dr.  Neil  Eastman,  Mr.  Joseph  Fox,  Dr. 
Ugo  Gagliardi,  Dr.  Leonard  Haynes,  Dr.  Ed  Lieblein,  and  Dr.  Edith  Martin.  In  addition, 
the  panel's  preliminary  conclusions  were  presented  to  a  Defense  Science  Board  Task  Force 
on  October  23,  1985. 

While  the  panel  found  a  number  of  major  problems,  it  should  be  emphasized  that 
the  panel  felt  that  the  19  September  STARS  Program  Plan  and  the  present  state  of  the 
STARS  Program  were  adequate  starting  points  for  arriving  at  a  desirable  and  beneficial 
STARS  Program. 

The  following  is  a  list  of  the  major  problem  areas  and  recommendations: 

( 1 )  The  STARS  Program  cannot  succeed  with  the  current  committee 
management  approach.  A  strong  centralization  of  authority  is  needed  with 
associated  spending  and  contracting  capabilities. 

(2)  The  STARS  Program  needs  a  concrete  vision  of  how  the  defense 
community  will  operate  in  the  mid- 1 990’s  along  with  an  appropriate 
unifying  theme  outlining  the  contribution  of  STARS  in  achieving  that 
vision. 

(3)  The  current  STARS  theme  is  focused  on  developing  improved  software 
environments  and  promoting  their  widespread  use.  A  major  problem  with 
this  theme  is  that  it  does  not  lay  the  foundation  for  continued  improvement 
after  STARS. 

(4)  A  new  STARS  theme  is  needed  which  focuses  on  exploiting  the  forces  in 
the  marketplace  and  directs  Program  activities  on  improving  DoD's  ability  to 
buy  products  from  the  marketplace. 

(5 )  Action  must  be  taken  to  accelerate  program  execution  Planned  activities 
that  are  consistent  with  this  marketplace  theme  should  be  continued  or 
hastened  Some  activities  should  be  adjusted  and  some  terminated 


(6)  The  panel  assessed  each  of  the  six  technical  areas:  ( 1 )  Software 
Engineering  Environments  (SEE),  (2)  Methodology,  (3)  Application 
Specific,  (4)  Business  Practices,  (5)  Human  Resources,  and  (6) 
Measurement  Problems  concerning  incompleteness,  inconsistency,  and 
lack  of  coordination  were  found.  Possibilities  for  improvement  which  are 
consistent  with  the  marketplace  theme,  have  been  outlined.  The  Business 
Practices  Area  is  viewed  as  the  most  critical  of  the  six  areas. 

(7)  Attention  must  be  rapidly  brought  to  bear  on  obtaining  intensive  industry 
involvement  in  STARS  activities. 

Each  of  these  points  is  discussed  below. 

2.0  MANAGEMENT  ISSUES 

2.1  Problem  Areas 

The  STARS  Program  has  been  crippled  by  serious  political,  organizational,  and 
management  problems  that  must  be  resolved  before  any  real  progress  in  the  substantive 
areas  can  be  expected. 

At  all  levels,  the  STARS  Program  has  been  subjected  to  management  by  committee. 
At  the  top  level  of  the  organization,  there  is  an  OSD/Tri-Service  committee  arrangement  that 
has  resulted  in  sometimes  paralyzing  conflicts  over  the  distribution  of  authority  and 
responsibility  for  the  program.  These  conflicts  have  had  serious  consequences  on  program 
planning  and  execution  and  have  drained  the  program  of  significant  management  attention 
and  energy.  Program-level  decisions  are  being  made  by  Service  Managers  acting  as  a 
committee  with  most  recommendations  coming  from  the  bottom-up  The  six  areas  are 
being  planned  and  managed  by  committees  consisting  largely  of  non -dedicated  personnel 
from  service  laboratories. 

The  lack  of  an  effective  hierarchical  organization  within  the  SJPO  has  acted  as  a 
further  hindrance.  Under  the  current  organization,  the  STARS  Director  has  no  line 
authority  over  the  Service  Program  Managers,  their  Deputies,  or  the  leaders  of  the  six  Area 
Coordinating  Teams  (ACTs).  With  the  exception  of  the  Technical  Director,  all  STARS 
personnel  report  to  and  are  evaluated  by  people  from  outside  of  the  SJPO.  Clearly,  this 
organization  works  against  any  kind  of  centralized  program  management.  The  lack  of  an 
effective  management  organization  is  paralleled  by  the  lack  of  an  effective  executing 
organization.  In  particular,  significant  delays  have  occurred  in  transferring  funds  and 
initiating  contracts. 

2.2  Panel  Recommendations 


The  areas  requiring  immediate  attention  are  those  involving  the  establishment  of 
clear  lines  of  authority  and  accountability  as  well  as  effective  mechanisms  for  spending  and 
contracting.  The  panel  strongly  recommends  the  following  actions  be  taken  immediately 

( 1 )  Appoint  and  empower  a  STARS  Program  Director.  The  most  immediate 
requirement  is  to  identify,  hire,  and  empower  a  strong  permanent  Director  and 
to  establish  a  tenable  political  situation  for  that  Director  The  STARS  Director 
must  be  given  control  of  the  program 

(2)  Resolve  the  contention  between  QSD  and  the  Services.  The  current  contention 
between  OSD  and  the  Service  Program  Managers  over  who  is  in  control  of  the 
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STARS  Program  must  be  resolved  quickly  if  the  STARS  Program  is  to  survive. 
This  resolution  will  require  management  attention  at  levels  above  CSSD  and 
SJPO.  This  may  require  reprogramming  the  STARS  budget  as  an  OSD 
element. 


(3)  Give  the  STARS  Director  spending  and  contracting  authority.  The  STARS 
Director  must  be  given  effective  spending  and  contracting  authority.  The 
current  checks-and-balances,  distributed  consensus  approach  to  spending 
authority  has  not,  and  will  not,  work. 


(4)  Form  an  effective  line  management  organization.  A  line  organization  with  clear 
execution  authority  and  accountability  is  critical  to  the  Program's  success. 
Several  alternatives  are  suggested  in  Volume  II. 

(5)  Establish  effective  administrative  procedures.  Effective  administrative 
procedures  must  be  established  for  contracting,  tracking  disbursements, 
preparing  plans,  reviewing  plans,  and  reviewing  accomplishments. 


and  advisory  in  nature  rather  than  managerial. 
3.0  VISION  OF  THE  RESULT  OF  STARS 


i.  ACTs  should  be  technical 


The  ultimate  goal  of  the  STARS  Program  must  be  to  ensure  DoD's  the  ability  to 
build  and  support  the  mission  critical  systems  of  the  early  1990's.  While  doing  this,  the 
basis  must  be  established  for  meeting  software  requirements  in  the  years  beyond  this 
century.  STARS  is  intended  to  be  a  program  of  finite  length,  ending  with  the 
institutionalization  of  the  improvements  it  has  fostered,  and  of  the  processes  leading  to 
further  improvement. 


In  reviewing  the  September  19  Program  Plan,  the  panel  concluded  that  the  current 
goals  of  the  STARS  Program  are  sound  and  important  The  goals  have  remained  largely 
unchanged  since  the  pre-Raleigh  period  and  have  emerged  as  a  result  of  intense  critical 
examination  and  consensus  building  from  government,  industry  and  academia.  Beyond 
general  and  abstract  goals,  STARS  needs  to  operate  within  the  context  of  a  concrete  vision 
of  how  the  defense  community  should  develop  and  support  MCCR  software  in  the  early 
1900's.  The  panel  believes  that  the  vision  should  be  along  the  lines  of  that  discussed 
below. 


The  defense  community  will  be  well  ahead  of  potential  adversaries,  rapidly 
improving,  and  even  widening  the  gap  between  them.  Defense  software  will  be  meeting  its 
requirements  with  the  needed  quality,  on  time,  for  reasonable  cost,  and  doing  so 
predictably.  To  do  this,  technology  will  be  flowing  smooothly  into  use  in  the  U.S.  defense 
community  regardless  of  its  commercial,  academic,  or  government  origin.  Facilitating  this 
flow  will  be  a  regulatory  context  that  attracts  the  needed  investments  by  providing  equitable 
return  for  all  parties  (including  the  Government)  and  a  marketplace  for  the  resulting 
technology,  particularly  as  embodied  in  software  tools  and  in  reusable  MCCR  software. 

The  DoD  will  be  an  intelligent  buyer  of  software  and  investor  in  software 
technology.  The  marketplace  will  be  broad  and  the  Government  will  be  able  to  obtain  the 
best  software  and  software  technology,  possibly  obtaining  different  pieces  or  services  from 
different  vendors. 


Every  project  will  readily  be  able  to  establish  an  appropriate  environment,  or  rapidly 
modify  its  existing  environment  to  meet  the  needs  of,  and  constraints  upon,  the  project 

It  w  .11  be  possible  to  rapidly  produce  initial  and  subsequent  versions  of  MCCR 
software.  1  ie  productivity  and  quality  will  be  greatly  improved  over  the  present  MCCR 
software  levels. 

Key  to  the  functioning  of  this  marketplace  will  be  standardized  interfaces  among 
environments  that  allow  transfer  of  (at  least)  workproducts  and  (at  most)  all  types  of 
software  tools  and  users.  The  compatibility  provided  by  these  interfaces,  along  with  the 
use  of  suitable  acquisition  practices  by  the  DoD,  provides  the  bases  for  the  success  of  the 
marketplace  and  the  cumulative  improvement  of  the  community,  allowing  a  competitive 
advantage  to  be  reached  by  innovation  rather  than  by  "reinventing  the  wheel." 

The  ability  to  transfer  software  among  Software  Engineering  Environments  (SEE's) 
is  additionally  important  to  DoD  because  of  the  need  to  transfer  software  for  logistics  and 
because  the  reuse  of  software  among  DoD  programs  potentially  provides  great  cost 
effectiveness. 

4.0  STARS  THEME 

This  vision  of  the  result  of  STARS  assumes  the  creation  of  a  marketplace  of 
competitive  suppliers  of  methods,  tools,  MCCR  software  components,  and  related 
technologies.  It  also  assumes  that  DoD,  in  return,  will  be  an  intelligent  consumer  of  those 
products.  The  panel  feels  strongly  that  STARS  should  adopt  and  consistently  operate 
within  this  marketplace  theme. 

The  marketplace  theme  can  be  contrasted  with  the  current  theme  of  the  STARS 
Program  emphasizing  the  development  of  improved  software  environments.  The  main 
problem  with  this  theme  is  that  it  focuses  on  the  end-products  to  be  achieved  rather  than  on 
capturing  a  process  for  continued  improvement.  The  result  is  an  increment  in  capability 
without  any  provisions  for  further  improvements. 

In  contrast,  the  theme  of  creating  and  using  marketplaces  is  important  because  it 
lays  the  foundation  for  continued  improvement  after  the  inital  improvements  resulting 
directly  from  the  STARS  Program.  It  docs  this  by  attracting  and  leveraging  investment 
from  the  private  sector,  by  fostering  a  high  degree  of  portability  for  software  tools,  by 
speeding  the  flow  of  technology  into  widespread  use,  and  by  supporting  the  reuse  of 
software  system  components. 

The  marketplace  theme  is  inherently  a  more  difficult  theme  than  one  in  which 
STARS  simply  pays  for  the  development  of  new  tools,  systems,  and  related  technology 
without  considering  how  these  developments  will  be  used  or  how  they  will  reduce  costs  in 
the  future.  Every  project  area  undertaken  by  STARS  will  face  the  difficult  decision  of  how 
much  STARS  must  fund  in  order  to  ensure  that  a  given  capability  will  appear  in  the 
marketplace  at  the  appropriate  time.  Must  some  concepts  be  demonstrated  as  feasible  in 
order  to  encourage  investment  from  the  private  sector?  Must  the  value  of  some  concepts  be 
demonstrated  to  potential  users?  Must  certain  items  be  produced  or  purchased?  What 
standards  are  required  and  when?  What  new  business  practices  within  DoD  are  needed  to 
establish  the  consumer  and  producer  side  of  a  given  market?  These  kinds  of  decisions  will 
require  close  cooperation  with  industry  and  a  keen  awareness  of  political,  entrepreneurial, 
and  technical  issues  by  the  STARS  leadership. 
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It  is  important  to  keep  in  mind  that  there  are  two  marketplaces  with  which  STARS 
must  be  concerned.  One  is  the  marketplace  for  software  engineering  tools  and  methods;  the 
other  is  the  marketplace  for  MCCR  software  components. 

Creating  a  marketplace  in  software  engineering  tools  and  methods  requires  solving 
several  problems.  Chief  among  them  is  providing  for  technical  compatibility  through 
standardized  interfaces.  Other  key  issues  are:  (1)  attracting  investment  from  the  private 
sector  (being  careful  not  to  exclude  small  innovative  firms),  (2)  instituting  appropriate 
acquisition  policies/practices,  and  (3)  providing  a  comparative  evaluation  capability. 

Creating  a  marketplace  in  MCCR  software  components  requires  solving  many  of 
the  same  problems  that  occur  in  the  software  engineering  tools  marketplace.  In  addition,  a 
number  or  non-technical  issues  make  the  development  of  this  marketplace  complex, 
especially  as  it  relates  to  reusable  software.  These  issues  include: 

(1)  Classified  software 

(2)  Proprietary  interests 

(3)  Rights  in  data  (e.g.,  derivative  works) 

(4)  Undesirable  foreign  technology  transfer 

(5)  Libraries,  warehousing,  cataloguing  and  retrieval 

(6)  Support 

(7)  Incentives 

(8)  Royalties 

(9)  Making  software  a  recurring  business 

Many  of  these  issues  have  received  attention  within  the  STARS  Program  but  much  more 
effort  is  required  before  they  are  entirely  solved.  The  Business  Practices  and  Application- 
Specific  Areas  will  be  particularly  critical  in  this  regard.  A  number  of  marketplace  issues 
are  discussed  in  Volume  II. 

The  results  of  an  assessment  of  each  of  the  STARS  technical  areas  is  contained  in 
the  following  section. 

5.0  STARS  AREAS 

5.1  General  Issues 

The  STARS  Program  is  currently  divided  into  six  areas: 

(1)  Business  Practices 

(2)  Human  Resources 

(3)  Software  Engineering  Environments  (SEE) 


(4)  Application  Specific 

(5)  Measurement 

Each  of  these  areas  is  represented  by  an  Area  Coordinating  Team  (ACT)  staffed  largely  by 
personnel  from  DoD  laboratories.  Across  areas,  the  teams  vary  widely  in  terms  of  the 
technical  competency  of  their  membership.  The  panel  recommends  that  competency 
requirements  be  established  for  team  membership  and  that  the  ACTs  include  experts  from 
outside  DoD  as  well.  The  role  of  the  ACTs  should  be  to  serve  as  technical  advisors  to 
STARS  management  rather  than  their  current  role  as  project  managers. 

If  the  theme  of  the  STARS  Program  shifts  away  from  the  current  emphasis  on 
developing  production-quality  environments  to  an  emphasis  on  fostering  a  competitive 
marketplace,  there  will  be  a  corresponding  shift  in  the  relative  importance  of  the  STARS 
areas.  Under  the  marketplace  theme,  the  Business  Practices  Area  is  the  most  important 
because  of  its  concern  with  issues  involved  in  stimulating  the  marketplace  and  defining  the 
context  for  how  DoD  should  conduct  business  within  that  framework.  Under  the  current 
theme,  the  lion's  share  of  the  funding  and  attention  is  going  into  the  SEE  Area.  Under  the 
marketplace  theme,  the  SEE  Area  is  still  important  but,  as  discussed  below,  its  role  is  to 
provide  the  compatibility  framework  and  to  undertake  prototype  developments  rather  than 
to  build  production-quality  environments. 

The  emphasis  of  the  other  areas  changes  as  well  under  the  marketplace  theme.  The 
Human  Resources  Area  takes  on  the  difficult  role  of  educating  DoD  personnel  to  be  an 
intelligent  consumer  in  the  new  marketplace  and  the  task  of  discovering  marketplace  stimuli 
for  the  improvement  of  software  engineering  capabilities  in  the  work  force.  The 
Measurement  Area  is  important  because  of  the  need  to  objectively  and  quantitatively 
evaluate  competing  products  and  because  of  the  need  to  assess  the  impact  of  STARS.  The 
Application-Specific  Area  can  serve  to  reduce  risk  for  the  DoD  program  manager  by 
providing  feasibility  demonstrations  which  involve  building  MCCR  systems  from 
marketplace  components.  Finally,  the  Methodology  Area  can  provide  the  capability  for 
evaluating  and  selecting  from  methods  and  supporting  tools  in  the  marketplace. 

The  panel  feels  that  the  current  set  of  six  areas  may  not  represent  the  optimal  set  for 
STARS  under  a  marketplace  theme.  However,  the  panel  also  feels  more  critical  problems 
must  be  solved  immediately  in  the  realm  of  program  management  and  execution.  Any  kind 
of  restricting  of  areas  can  wait  until  the  Program  is  moving  forward. 

Regardless  of  which  theme  is  adopted,  there  exist  problems  of  inconsistency, 
incompleteness,  and  lack  of  coordination  across  the  areas.  Each  of  the  areas  is  discussed  in 
more  detail  below. 

5.2  Business  Practices  Area 

The  Business  Practices  Area  has  been  severely  underemphasized.  Its  original 
purpose,  as  presented  at  the  initial  public  review  held  at  Raleigh,  was: 

( 1 )  To  develop  the  body  of  knowledge  necesary  for  the  DoD  to  act  in  the  role  of 
intelligent  consumer. 

(2)  To  develop  a  suite  of  program  management  tools  to  provide  consistency  and 
support  across  program  phases  and  across  programs  and  application  areas. 
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(3)  To  propose  changes  to  DoD  regulations,  practices,  and  standards  appropriate  to 
enhancing  the  DoD  position  in  the  business  environment 

More  recently,  this  Area  has  narrowed  its  focus  on  the  acquisition  of  software  and 
eliminated  any  emphasis  on  developing  the  knowledge  to  improve  and  exploit  the  DoD 
position  in  the  commercial  sector. 

STARS  must  recognize  that  the  Business  Practices  Area  is  the  most  powerful  tool 
that  the  Program  has  to  insert  appropriate  software  technology  into  the  DoD  community.  It 
is  also  the  most  difficult  area  in  which  to  accomplish  anything  significant.  The  current  level 
of  funding  does  not  reflect  this  and  should  be  increased  drastically. 

The  Business  Practices  Area  should  be  concerned  with  the  steps  needed  to  stimulate 
the  supplier  side  of  the  marketplace.  One  of  the  critical  factors  determining  the  success  of 
the  marketplace  strategy  is  the  extent  to  which  investment  can  be  attracted  from  the  private 
sector.  Investment  behavior  is  the  determined  by  degree  of  uncertainty  or  risk.  DoD  can 
help  to  reduce  risk  in  two  important  areas:  one  is  in  conducting  R&D  to  demonstrate 
feasibility  and  the  other  is  to  remove  the  uncertainty  from  its  own  future  market-related 
actions  and  policies. 

One  of  the  goals  of  the  STARS  Program  should  be  to  attract  investment  and 
participation  from  a  number  of  smaller  firms  from  which  one  would  expect  a  greater 
likelihood  of  high-quality,  innovative  products.  Small  firms  can  be  attracted  if  they  can 
enter  the  market  with  a  product  requiring  relatively  modest  capital  investment  that  will  work 
compatibly  with  other  products  in  the  marketplace.  This  is  one  of  the  primary  reasons  that 
a  compatibility  framework  is  needed.  In  addition,  the  larger  the  market,  the  more  attractive 
it  will  be  to  large  and  small  investors.  Technical  compatibility  beyond  just  the  DoD 
community  can  thus  be  beneficial  to  DoD.  These  types  of  issues  need  to  be  addressed  by 
the  Business  Practices  Area  and,  in  this  instance,  implemented  by  the  SEE  Area. 

If  STARS  is  to  influence  the  marketplace,  one  of  the  first  important  steps  will  be  to 
identify  and  foster  cooperative  relationships  with  the  potential  players  on  the  supply  side. 

At  a  minimum  these  players  include  the  following: 

(1)  The  major  hardware  manufacturers  (e.g.,  IBM,  DEC) 

(2)  System  houses  (e.g.,  Lockheed,  General  Electric) 

(3)  Software  houses  (e.g.,  CSC) 

(4)  Small  start-ups  whose  future  lies  in  high  quality  and  innovation 
(e.g.,  Rational,  Verdix) 

(5)  Standards  and  other  professional  organizations  (e.g.,  ANSI,  IEEE,  EIA) 

Clearly,  the  Business  Practices  ACT  will  require  people  with  the  needed  expertise 
in  marketplace  issues,  including  lawyers  and  executives  from  the  private  sector.  The 
Business  Practices  ACT  must  be  closely  coordinated  with  the  remaining  areas.  In  many 
cases,  this  area  will  drive  the  activities  of  the  other  areas. 


5.3  Human  Resources  Area 


The  efforts  within  the  Human  Resources  Area  are  divided  into  three  activities.  The 
first  two  try  to  establish  the  size  of  the  current  workforce  and  the  workforce  of  the  1990's. 
The  third  concerns  development  of  material  and  technology  for  education  and  training. 

Within  the  marketplace  theme,  this  area  could  have  an  additional  job  that  will  be 
much  more  challenging  than  just  that  of  creating  a  workforce  doing  a  "better"  job  of 
business  as  usual.  Instead,  the  Human  Resources  Area  could  face  the  task  of  transforming 
the  DoD  into  a  consumer  with  the  capability  of  evaluating  and  selecting  from  the 
marketplace  in  an  intelligent  manner.  This  would  represent  a  very  different  way  of 
conducting  business  and  no  less  than  a  pervasive  cultural  change. 

The  major  barriers  to  the  success  of  the  STARS  Program  will  be  political, 
organizational,  social,  and  psychological  even  more  than  technical.  Changing  the  behavior 
of  individuals  and  organizations— especially  those  that  are  comfortable  with  die  current 
way  of  doing  business — can  be  expected  to  meet  with  varying  degrees  of  resistance  from 
many  sources.  If  any  area  is  capable  of  addressing  these  "people  issues,"  it  should  be  the 
Human  Resources  Area.  This  will  involve  the  ability  to  anticipate  these  sources  of 
resistance  as  well  as  likely  advocates  and  formulate  a  strategy  which  takes  both  into 
account 

5.4  Software  Engineering  Environments  (SEE)  Area 

The  SEE  Area  is  currently  the  major  focus  of  attention  of  the  STARS  Program. 

The  panel  views  it  as  overemphasized  both  in  management  attention  and  in  funding  during 
the  year  preceding  September  1985.  The  panel  feels  that  the  strategy  for  the  SEE-related 
activities  presented  in  the  19  September  STARS  Program  Plan  is  too  oriented  toward 
building  environments  as  opposed  to  stimulating  the  marketplace  to  provide  the 
environments. 

The  major  contribution  of  the  SEE  Area  should  be  a  framework  for  constructing 
SEE's  out  of  pieces  or  fragments.  One  motivation  for  doing  this  is  to  encourage 
participation  by  smaller  firms  who  cannot  afford  to  invest  in  entire  toolsets  or  environments 
but  can  tackle  one  piece,  with  a  substantial  liklihood  of  doing  that  one  piece  very  well.  The 
panel  proposes  the  following  as  the  acquisition  strategy  for  the  SEE  which  has  the 
advantage  of  providing  a  general  framework  for  environments  onto  which  generic  tools 
(e.g.,  editors,  compilers)  can  be  added  as  well  as  application-specific  tools.  A  key  aspect 
of  this  strategy  is  to  shift  attention  away  from  developing  complete,  wide-spectrum 
environments  (at  least  under  Government  sponsorship)  and  toward  the  production  of 
compatible  toolsets. 

The  suggested  approach  is  to  interact  with  the  commercial  software  development 
marketplace  as  an  intelligent  and  powerful  consumer.  In  order  to  stimulate  the  commercial 
sector  to  provide  appropriate  and  sufficient  responsiveness,  the  STARS  Program  must  take 
four  separate  but  related  actions.  These  are: 

(1)  Reduce  the  risk  of  commercial  participation  by  demonstrating  technical 
feasibility  through  the  development  of  necessary  prototypes.  These 
prototypes  will  be  done  commercially  and  made  commercially  available. 
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(2)  Provide  standards  and  guidlines  for  the  development  of  tools,  both  prototype 
and  production  quality,  that  will  be  used  in  the  toolsets. 

(3)  Announce  an  intention  to  acquire  tools  developed  under  commercial-sector 
support  to  be  used  in  the  toolsets. 

(4)  Modify  government  acquisition  regulations  and  propose  legislation,  if 
neccessary,  to  allow  commercial  ownership  of  most  of  the  products  developed 
under  this  strategy. 

The  actual  development  and  acquisition  would  follow  a  simple  tactical  plan  which  is 
outlined  in  Volume  II. 

5.5  Application  Specific  Area 

Many  of  the  problems  outlined  earlier  in  connection  with  creating  a  marketplace  of 
MCCR  software  components  are  most  appropriately  addressed  by  the  Application-Specific 
Area,  particularly  those  involved  in  warehousing,  cataloguing,  and  retrieving  components. 
Another  important  contribution  of  this  area  should  be  feasibility  demonstrations  within 
specific  application  areas  as  a  risk-reduction  vehicle  for  DoD  program  managers. 

The  activities  currently  planned  within  the  Application  Specific  Area  are  largely 
unrelated  and  disconnected  from  other  areas.  The  former  is  not  too  surprising  and  may  be 
justified  by  the  need  to  demonstrate  STARS  in  diverse  applictions.  It  is  important, 
however,  that  links  with  other  STARS  areas  and  other  programs  be  institutionalized.  There 
are  some  common  aspects  to  the  activities  across  areas  and  it  is  not  clear  that  there  is  an 
activity  that  will  coalesce  the  common  experiences  and  adequately  transfer  the  new 
knowledge. 

5.6  Measurement  Area 

The  current  Measurement  Area  Plan  (4  November  1985)  contains  some  useful  and 
needed  activities,  especially  those  involved  in  providing  standard  definitions  for  various 
measures  and  in  developing  a  central  database  of  measurement  data.  But  there  are  also 
activities  such  as  tool  building,  model  development  and  calibration,  and  the  development  of 
training  materials  which  are  unlikely  to  increase  the  use  of  quantitative  techniques  because 
the  incentives  for  that  use  are  missing. 

The  primary  aspect  of  the  Measurement  Area  is  to  play  a  pivotal  role  in  the 
marketplace  strategy.  Tools,  methods,  and  other  products  will  be  successful  in  the 
marketplace  only  if  they  are  of  production  quality  and  fully  supported.  The  consistent  use 
of  metrics  across  the  community  will  allow  developers  and  the  entire  market  to  judge  new 
products  as  well  as  to  compare  experience  with  new  and  older  products.  This  will  require 
an  evaluation  capability  that  relies  on  measurable  characteristics,  helps  in  certifying  new 
marketplace  entries,  and  assists  in  comparing  among  the  various  alternatives  that  will 
emerge.  The  panel  feels  that  competent  technical  people  in  each  activity  area  are  the 
appropriate  ones  to  formulate  approaches  to  this  aspect  of  measurement  The  Measurement 
Area  Coordinating  Team  can  serve  a  vital  role  in  ensuring  consistency  in  the  measures  and 
measurement  procedures  used  across  areas. 

The  secondary  aspect  of  the  Measurement  Area  should  be  to  assess  the 
effectiveness  of  the  STARS  Program.  The  Measurement  ACT  must  work  with  each  team 
and  with  STARS  management  to  establish  measurable  goals  and  suitable  mechanisms  to 
determine  success. 
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5.7  Methodology  Area 


The  Methodology  Area  addresses  the  problem  of  providing  a  variety  of  software 
creation  and  evolution  approaches  for  use  on  DoD  software  projects.  The  overall  intent  is 
to  broaden  both  the  applicability  of  the  collection  of  methods  available  for  use  and  the 
extent  to  which  they  are  actually  used. 

The  activities  of  this  area  will  result  in  the  enhancement  of  existing  methods  and  the 
development  of  new  ones  if  warranted.  The  major  focus  of  the  area  is,  however,  on 
developing  capabilities  for  classifying,  evaluating,  and  selecting  methods,  a  focus  which  is 
consistent  with  the  marketplace  theme.  These  capabilities  will  provide  direct  help  in  the 
choice  of  a  method  to  support  a  specific  project  Development  of  the  capabilities  will  help 
in  identifying  enhancement  and  additions  to  existing  methods. 

The  area’s  plans  generally  seem  adequate  for  the  near  term.  However,  much  more 
concrete  plans  are  needed  for  the  medium-to-long  term. 

5.8  Additional  Areas  for  Reconsideration 

5.8.1  Systems  Area 

Systems  Area  was  one  of  the  original  STARS  areas  at  the  time  of  the  Raleigh 
workshop.  The  panel  feels  that  the  content  of  this  area  is  of  increasing  importance  to  DoD 
and  should  be  adequately  covered.  A  new  "Systems"  initiative/program,  bridging  STARS 
and  VHSIC,  may  be  appropriate.  In  any  case,  STARS  should  include  software-related 
systems  concerns.  For  example,  the  emphasis  on  reliability /fault  tolerance  in  the  original 
Systems  Area  should  not  be  lost. 

5.8.2  Human  Engineering  Area 

Human  Engineering  was  also  one  of  the  original  STARS  areas  at  the  time  of  the 
Raleigh  workshop.  It  was  broad  in  its  concerns,  covering  not  only  human  engineering  of 
the  SEE  (including  methods  as  well  as  tools)  but  also  human  engineering  of  MCCR 
systems  for  the  end  user.  Human  Engineering  disappeared  as  a  separate  area  of  the 
STARS  Program  in  1984. 

Human  Engineering  should  be  reinstated  as  an  identifiable  area.  It  requries 
expertise  which  lies  more  in  the  domain  of  the  behavioral  sciences  than  in  computer 
science.  Thus,  it  is  unlikely  to  fall  out  of  any  of  the  other  areas. 

Consistent  with  the  marketplace  theme,  the  Human  Engineering  area  should  address 
the  issues  involved  in  specifying,  evaluating,  and  selecting  products  of  the  basis  of  their 
usability.  The  area  should  also  focus  on  the  issues  involved  in  establishing  information 
interfaces  to  support  independence  between  a  user  interface  and  the  underlying  tools  or 
applications. 

6.0  SUMMARY 

The  most  critical  problem  facing  the  STARS  Program  is  managerial  and  must  be 
resolved  immediately.  The  panel  feels  strongly  that  once  this  problem  is  addressed, 

STARS  should  adopt  and  consistently  operate  within  a  theme  of  marketplace  stimulation. 
The  marketplace  theme  is  important  because  it  leads  to  direct  and  extensive  industry 
involvement  in  STARS.  By  developing  products  under  private-sector  investment  and 
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letting  the  investors  retain  proprietary  rights,  industry  should  actively  participate  in 
achieving  the  Program's  goals  and  objectives,  with  numerous  benefits. 

The  strengths  of  the  American  free-enterprise  system — including  private  initiative, 
competition,  and  the  desire  to  make  a  profit— should  be  exploited.  Industry  understands 
the  need  to  supply  a  fully-supported  product  and  is  motivated  to  get  its  products  used.  A 
marketplace-based  strategy  will  therefore  foster  both  follow-on  support  for  products  and  a 
technology  push  from  within  industry.  In  addition,  expertise  other  than  that  directly 
contracted  for  will  be  attracted  to  the  solution  of  DoD  problems. 

Creating  a  marketplace  of  software  engineering  methods  and  tools  and  a 
marketplace  of  MCCR  software  components  will  require  developing  a  customer  base, 
developing  a  base  of  private  investment,  and  developing  a  facilitating  regulatory  context. 
The  marketplace  theme  is  inherently  a  riskier  alternative  than  the  current  approach  of 
directly  paying  for  the  construction  of  several  production-quality  environments.  The 
benefits  are  many  times  greater,  not  only  for  DoD,  but  for  the  nation  as  a  whole  because  the 
foundation  will  be  in  place  for  the  cummulative  improvement  of  the  entire  industry. 

STARS  must,  however,  be  sensitive  to  the  increased  risk,  both  from  the  consumers' 
perspective  and  from  the  suppliers'. 

The  panel  recommends  that  STARS  management  take  a  new  look  at  the  set  of  areas 
and  their  relative  emphasis  within  the  Program.  The  panel  recommends  that  the  superceded 
Systems  and  Human  Engineering  Areas  be  reconstituted.  Having  reviewed  the  areas  and 
their  relative  levels  of  emphasis,  management  should  review  the  constitution  of  the  area 
teams  and  repeat  this  process  periodically.  The  focus  of  the  teams  should  be  changed  from 
a  managerial  to  a  technical  orientation;  management  should  be  handled  at  higher  levels. 
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The  key  to  the  new  marketplaces  will  be  technical  compatibility 
provided  by  interface  standards 


Marketplace  theme  is  more  difficult  and  involves  greater  risk. 
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-  libraries,  warehousing,  cataloguing,  retrieval 

-  support 

-  incentives 

-  royalties 

-  making  software  a  recurring  business 
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